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EXECUTIVE  SUMMARY 


The  purpose  of  this  study  is  to  review  the  capabilities  of 
the  risk  analysis  technique,  VERT  (Venture  Evaluation  and  Review 
Technique)  and  to  discuss  its  recent  applications  to  assist 
Program  Managers.  Information  for  the  preparation  of  this 
report  was  primarily  obtained  through  informal  telephone  in- 
terviews with  personnel  who  have  conducted  VERT  analyses. 

VERT  uses  a simulation  approach  to  analyze  decision  situa- 
tions involving  risk.  The  model  is  easy  to  use  and  its  variety 
of  program  functions  allows  analysis  of  complex  decisions 
involving  many  interrelated  risk  situations  (activities). 

This  relatively  new  technique  has  been  applied  in  a broad 
range  of  program  development  situations.  This  report  discusses 
some  applications  in  support  of  Army  and  Navy  programs.  These 
applications  included  analysis  of  develooment  schedules,  pro- 
duction costs  and  schedules, and  system  configuration  alternatives. 
In  most  applications,  the  major  concerns  were  cost  and  schedule 
risks.  Analyses  where  technical  performance  risk  is  the  major 
concern  have  been  limited. 

In  addition  to  its  risk  analysis  function,  VERT  is  being 
used  to  perfomn  sensitivity  analyses  and  in  one  case  to  track 
an  acquisition  program's  progress.  In  these  two  applications 
VERT  provides  the  decision  maker  with  information  which  can  be 
a great  benefit  in  answering  ’Vhat  if"  questions. 
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The  report  ooneludea  that  VERT  la  a powerful  and  flexible 
tool  which  can  be  of  great  aaaiatanee  to  deciaion  makera. 

However,  the  capabilitiea  of  VERT  are  not  well  known,  and  con- 
sequently the  potential  of  this  technique  ia  not  being  realized 
within  the  aystema  acquisition  community.  Further  investigation 
into  the  current  uses  of  VERT  and  suggestions  for  increased 
efforts  to  inform  program  management  persoraiel  about  VERT  are  some 
recommendations  resulting  from  this  study. 
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CHAPTER  I 


INTRODUCTION 

Background 

One  of  the  most  pervasive  and  recurring  situations  encoun- 
tered in  Program  Management  is  the  requirement  to  make  decisions 
with  incomplete  or  inadequate  information  about  the  alternatives. 
These  "decisions  under  risk"  situations  occur  more  frequently  in 
Weapons  Systems  Acquisition  Program  Management  than  in  many  other 
areas  of  management.  This  is  because  the  Program  Manager  (PM) 
is  involved  in  the  development  of  new  technologies  or  the  use 
of  existing  technologies  in  new  applications.  The  risks  en- 
countered by  the  PM  relate  to  the  throe  general  categories  of 
cost,  schedule  and  technical  performance. 

To  assist  the  PM  in  making  decisions  under  risk  many 
techniques  have  been  developed  and  used  over  the  past  years. 

Linear  Programming,  Game  Theory  and  various  modeling  techniques 
are  some  examples.  Cne  of  the  most  popular  modeling  approaches 
in  recent  years  for  complex  problems  has  been  the  math  modeling 
technique  called  "simulation."  The  easy  accessability  to  large 
scale  general  purpose  computers  has  made  this  technique  a valuable 
tool  for  the  PM. 

Purpose 

The  intent  of  this  report  is  to  discuss  and  to  some  degree 
comment  upon  recent  and  possible  future  uses  by  Defense  PM's  of 
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a simulation  risk  analysis  technique  called  VERT  (Venture 
Evaluation  Review  Technique ) (8 )^ . 

Ob.lectives 

The  objectives  of  this  study  were: 

1.  To  investigate  the  functional  capabilities  and  proce- 
dural requirements  of  VERT. 

2.  To  review  with  the  users  their  recent  attempts  and 
future  plans  to  use  VERT  as  a supportive  tool  in  the  decision 
process.  This  objective  was  oriented  to  identifying  what  types 
of  risk  were  considered  in  VERT  simulations  and  what  types  of 
problems  were  encountered  in  the  development  and  running  of  the 
VERT  model. 

3.  To  develop  a foundation  of  information  concerning  the 
VERT  model  and  identify  areas  for  further  investigation. 

4.  To  msQce  recommendations  concerning  the  future  use  of 
VERT  by  the  Systems  Acquisition  Community, 

Scope 

Because  of  time  and  travel  constraints  this  investigative 
effort  was  necessarily  limited  to  readily  accessible  documenta- 
tion and  informal  discussions  with  some  selected  personnel  who 
have  used  VERT  to  assist  PM's  in  decision  analyses.  Only 

^llliis  notation  will  be  used  throughout  the  report  for 
the  sources  of  quotations  and  major  references.  The  first 
number  is  the  source  listed  in  the  bibliography.  The  second 
number  is  the  page  in  the  reference  if  appropriate. 
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•’nilitary  organizations  have  been  contacted.  However,  since 
VERT  waa  developed  at  Rock  Island  Arsenal  Just  a few  years  ago. 
few  non-military  organizations  are  presently  aware  of  the  model. 

Definitions 

Model:  An  Imitation  or  artificial  representation  of  a real 
world  object  or  system  used  to  make  predictions  concerning  the 
performance  of  the  item  under  real  world  conditions. 

[ 

^ Simulation:  A stochastic  (proballstic ) modeling  technique 

which  provides  outputs  depicting  the  probability  distributions 

i- 

of  possible  outputs  expected  from  the  real  world  item. 

t Methodology 

This  study  effort  was  conducted  in  three  phases. 

First  the  functional  description  of  VERT  was  reviewed  by  the 
author  to  develop  an  appreciation  for  its  capabilities  and 
complexities. 

The  second  effort  involved  contacting  nine  known  users  of 
VERT  to  determine  their  willingness  and  availability  to  partici- 
pate in  an  informal  telephone  interview  discussing  their  experiences 
with  VERT.  Subsequently  an  outline  of  the  proposed  interviews 
was  developed  and  provided  to  six  individuals  prior ‘to  the 
conduct  of  the  interviews.  One  to  two  weeks  following  the 
forwarding  of  the  proposed  interview, the  interviews  were  con- 
ducted. 

The  final  phase  of  the  study  Involved  an  analysis  of  the 
data  collected  from  the  interviews  and  a survey  of  additional 
documentation  received  from  the  Defense  Logistics  Studies 
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CHAPTER  II 


THE  VERT  MODEL 


General  Description 

The  VERT  elmulation  model  uses  a general  networking  approach 

2 

to  structuring  risk  situations  to  be  analyzed  • This  approach 
allows  the  visual  depiction  of  all  Interrelated  actions  which 
may  occur.  Figure  1 shows  a very  general  network  depicting  a 
phase  in  the  development  of  a hypothetical  missile  system.  At 
certain  points  (nodes)  in  the  network,  decisions  are  made  which 
determine  the  next  path  to  be  followed  in  the  network.  The 
new  path  selected  at  these  decision  points  is  determined  by 
probabilities  or  some  criteria  specified  by  a mathematical  rela- 
tionship (8:9).  Although  this  (Pig.  1)  is  a very  simplified 
depiction  of  missile  system  development,  more  detailed  and 
complicated  situations  can  be  easily  described  by  this  type  of 
network. 

Once  the  network  is  developed  it  is  converted  to  the  VERT 
program  terminology.  The  program  has  a variety  of  input  capa- 
bilities which  allows  the  description  of  the  decision  events  and 
activities  occurring  in  the  network.  The  likelihood  of  any 

<^In  the  context  of  this  report,  risk  situations  are  those 
where  an  action  (decision)  may  cause  several  outcomes  each  of 
which  has  a certain  probability  of  occurrence. 
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activity  occurring  can  be  accurately  described  by  the  many 
common  forma  of  probability  distributions  included  in  the 
program  (See  incloaure  1).  In  addition,  the  relationship  of 
variables  such  as  time  and  cost  can  bo  defined  for  the  program 
as  a mathematical  relationship  (i,e,,  C = aT  + b), 

A final  characteristic  which  makes  VERT  a desirable  tool 
to  use  is  the  output  options  available.  The  program  provides 
distributions  depicting  the  frequency  at  which  certain  paths 
through  the  network  were  followed.  It  also  provides  distribu- 
tions showing  the  costs  and  time  involved  in  traversing  the 
different  paths. 

You  don't  have  to  be  a computer  programmer  or  systems 
analyst  to  effectively  utilize  VERT.  An  individual  in  a Program 
Office,  familar  with  basic  mathematics  and  computer  programming 
should  be  able  to  productively  use  VERT  after  several  hours  of 
study.  Complete  mastery  of  all  the  model's  capabilities  probably 
would  require  a week  of  continuous  effort  (8: (4.)  , However, 
such  proficiency  would  only  be  required  in  simulating  very 
complex  or  unique  risk  situations.  Actually,  even  in  the  most 
complex  situations,  the  programming  can  be  simplified  to  a great 
degree  by  breaking  down  the  network  into  subnetworks.  Then 
using  the  simple  program  functions  ^operators),  the  subnetworks 
can  be  simulated.  The  results  of  these  simulations  can  then  be 

i 

I used  as  Inputs  to  a higher  level,  more  general  network  simulation 

* which  ties  all  the  subnetworks  together.  There  may  be  some  loss 

I 

of  accuracy  in  the  final  results  from  this  two-step  approach, 
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However,  considering  the  chances  of  error,  and  the  time  involved 
in  solving  the  complex  network,  this  loss  of  accuracy  may  bo 
acceptable. 

In  contrast  to  VERT's  capability  to  simulate  complex  risk 
situations,  it  can  also  be  simplified  to  the  extent  that  it 
performs  the  same  general  functions  as  PERT  (Program  Evaluation 
Review  Technique )/C PM  (Critical  Path  Method).  By  setting  one 
of  the  VERT  decision  point  output  probabilities  to  1 and  the 
others  to  zero,  the  network  collapses  to  one  similar  to  a 
deterministic  PERT/CPM  network.  The  use  of  VERT  in  this  manner 
however  is  a waste  of  effort.  More  efficient  programs  exist 
for  the  analysis  of  networks  describing  situations  without 
risk. 

VERT  Procedures 

The  process  of  setting  up  and  conducting  a VERT  analysis 
can  be  summarized  into  five  basic  stops  which  many  times  may 
require  iterations  before  the  model  accurately  depicts  the  real 
world  situation. 

1.  The  first  effort  is  to  define  the  decision  (risk) 
situation  to  bo  analyzed  and  to  specify  the  objectives  of  the 
analysis . 

For  Instance  in  the  hypothetical  missile  system  mentioned 
earlier,  the  risk  situation  included  all  the  development  actions, 
the  decisions  which  were  to  be  made  and  their  interrelationship 
in  terms  of  time,  cost  and  technical  performance.  The  major 
objective  of  this  analysis  could  bo  to  determine  the  probability 
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of  going  into  production.  Impoaed  upon  this  could  be  the 
objectives  to  determine  the  cost  and  time  involved  in  reaching 
production.  In  addition,  the  decision  maker  may  want  to  know  the 
money  and  time  wasted  if  a decision  to  go  into  production  is  not 
made. 


The  situation  description  developed  during  this  step  should 
be  approved  by  the  decision  maker  before  proceeding  to  the 
following  steps. 

2.  In  this  step,  the  situation  defined  during  step  1 is 
depicted  in  a network.  This  visual  depiction  of  the  risk 
situations  helps  to  further  refine  the  description  developed  in 
the  first  step.  Errors  made  in  verbally  describing  the  situation 
often  become  evident  at  this  point.  Figure  1 is  an  example  of 

a descriptive  network  produced  during  this  effort. 

3.  Concurrently,  with  step  2,  the  data  necessary  to  des- 
cribe the  activities  and  decision  processes  should  be  collected. 
This  data  is  then  organized  into  some  form  of  a probability 
distribution  or  described  by  a mathematical  equation. 

4.  The  next  effort  is  to  transform  the  network  developed  in 
step  2 to  a network  which  will  facilitate  programming.  This 
involves  showing  the  decisions  and  activities  by  the  use  of  the 
logic  operators  of  the  VERT  program  and  the  conditions  developed 
in  step  3.  There  are  many  possible  solutions  to  this  transformed 
network  and  every  user  will  probably  accomplish  this  different- 
ly (8;4.)»  The  important  point  is  to  insure  it  accurately  des- 
cribes the  decision  process  of  the  original  network. 
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The  transformed  network  for  the  hypothetical  missile  system 
is  shown  in  figure  2,  A detailed  discussion  of  the  program 
operators  used  here  will  not  be  attempted  but  it  can  be  obtained 
from  the  author  (7:8-13).  However,  as  an  example  of  the  meaning  of 
the  transformation,  consider  the  notations  (AHD/ALL)  at  the 
beginning  of  the  "total  systems  test"  (figure  2).  These  operators 
require  all  the  Incoming  paths  to  have  been  successfully  completed 

I before  it  allows  the  simulation  to  perform  the  next  function. 

I In  the  missile  situation  it  means  that  total  system  test  will  not 

begin  until  all  subsystem  tests  are  satisfactorily  completed. 

The  activities  involving  risk  like  the  "development  of  a sub- 
system" show  reference  (A-3)  to  a table  of  probabilities  which  's 

will  be  used  in  the  program  to  describe  the  possible  outcomes 
of  the  effort. 

After  this  transform  is  complete,  the  information  is  entered 
in  a computer  program  and  the  simulation  is  run.  The  number  of 
iterations  or  xnins  of  the  program  is  determined  by  the  specified 
confidence  level  of  the  desired  results. 

5.  The  final  step  is  the  analysis  of  the  simulation  results. 

The  outputs  from  the  simulation  will  be  in  the  form  of  distribu- 
tions indicating  the  frequency  of  occurrence  of  the  different 
outcomes  or  the  distribution  of  costa  and  time  involved.  For 
instance  a cost  distribution  would  show  the  average  cost  and  the 

I 

range  of  values  which  might  occur.  Many  other  statistics  describ- 
ing the  simulations  operation  can  be  obtained  as  output  from  the 
program.  Analysis  of  all  the  output  information  will  provide  the 
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decision  maker  with  an  improved  understanding  of  the  risks 
involved  in  his  decision  situation. 

Another  benefit  can  be  realized  once  step  ^ has  been 
reached.  It  now  becomes  quite  easy  to  rerun  the  model  using 
variances  of  the  original  conditions.  This  form  of  sensitivity 
analysis  will  provide  the  decision  maker  with  answers  to  many 
of  the  "What  if"  questions.  Por  instance,  "What  if  the  subsystem 
test  of  the  air  vehicle  is  two  weeks  (average)  longer  than  origi- 
nally expected?"  A change  of  probably  one  punched  card  could 
be  made  and  the  entire  simulation  rerun  in  a very  short  period 
of  time.  The  new  outputs  would  provide  the  decision  maker  with 
good  indications  of  the  effects  of  this  hypothetical  contingency. 

In  summary,  the  VERT  model  is  flexible,  powerful  and  in 
most  cases  easy  to  use. 
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RECENT  PM  APPLICATIONS  OP  VERT 

Tha  VERT  modeling  technique  evolved  a few  years  ago  as 
an  Improvement  over  existing  risk  analysis  techniques  auch  as 
RISCA  (Risk  Information  System  for  Cost  Analysis  ( 6 )),  VERT 
is  an  improvement  over  most  exiating  models  in  three  areas. 

It  offers  a wider  range  of  functions  (decision  logic)  to 
describe  decision  actions  and  activities.  It  has  greater  inter- 
nal capability  for  specifying  the  probability  of  activities. 
Finally,  it  provides  increased  output  options  for  describing 
the  performance  of  the  simulation  model. 

This  technique  has  been  used  in  support  of  several  Army 
and  at  least  one  Navy  program.  In  the  following  discussion, 
a profile  will  be  developed  describing  these  current  apolications 
of  VERT.  The  majority  of  the  information  to  support  this  discus- 
sion was  obtained  by  informal  telephone  interviews  (see  biblio- 
graphy). 

Users 

Althou^  the  PM  is  the  ultimate  user  of  the  VERT  analysis, 
the  simulations  to  date  have  been  developed  and  run  by  functional 
offices  (normally  called  System  Analysis  Offices)  belonging  to 
the  Headquarters  supporting  the  PM.  In  all  but  one  case,  the 
offices  contacted  Indicated  they  were  sufficiently  staffed  with 
analysts  and  programmers  to  handle  their  present  risk-decision 
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analysis  requlroments.  In  addition  to  VERT,  these  offices  use 
other  risk  analysis  tools  such  as  RISCA  and  decision  trees. ^ 

One  systems  analysis  office  has  even  developed  a computer  program 
(DETRAK)  to  analyze  decision  trees  (10), 

The  PM's  who  have  received  VERT  simulation  support  vary 
from  the  major  programs  like  the  XM-1  tank  program  ( 3 ) snd 
the  AAH  (Advanced  Attack  Helicopter ) (13 ) program  to  the  small 
efforts  like  PEWS  (Platoon  Early  Warning  System)  ground  sensor 
project  (12).  VERT  has  also  been  used  by  a laboratory  in  the 
development  of  a new  muzzle  break  for  artillery  ( 5 ).  In  the 
Navy,  VERT  was  used  in  support  of  the  P-18  (Navy  fighter  airplane) 
program  (2 ) . ^ 

Applications 

To  date  the  majority  of  uses  of  VERT  have  been  concerned 
with  cost/schedule  risk  situations.  Only  one  of  the  individuals 
interviewed  had  attempted  to  use  VERT  in  estimating  technical 
risk. 

In  support  of  the  Cannon-Launched  Guided  Projectile  program 
a VERT  simulation  was  designed  to  examine  the  t5robabillty  that 
the  development  effort  would  successfully  reach  the  production 
phase.  Major  risks  considered  in  this  analysis  were  DT/OT  II 
and  III  and  the  probabilities  of  the  two  Low  Rate  Initial  Pro- 
duction (LRIP)  contractors  producing  hardware  acceptable  for 


^Decision  trees  are  networks  normally  used  to  analyze  decision 
situations  using  single  variables  (e.g,  cost)  and  exact  probabili- 
ties (versus  probability  distributions).  The  path  providing  the 
maximum  benefit  is  normally  selected. 


full  production.  Also  included  in  the  analysis  was  the  DSARC 
III  and  Ilia  decision  points  (1). 

In  another  application,  VERT  was  used  to  determine  the  cost/ 
schedule  risks  involved  in  achieving  the  planned  ICC  (Initial 
Operational  Capability)  for  the  MllOEl  self-propelled  howitzer 
using  a new  proppllant  charge  and  new  gun  tubes.  The  major 
risk  areas  considered  in  this  situation  were  manufacturing  prob- 
lems, acceptance  testing  (including  teat  firings)  and  shipping 
delays . 

In  the  case  of  the  AAH  program,  VERT  was  initially  used  to 
evaluate  the  planned  and  alternate  validation  phase  schedules 
through  DSARC  II.  Of  course,  at  this  stage  of  development 
there  is  considerable  risk  in  many  areas.  The  major  value  of 
this  analysis  would  be  to  show  all  the  possible  Impacts  from 
activities  which  may  not  occur  as  planned.  In  effect  it  allows 
you  to  determine  critical  paths  through  a network  full  of 
uncertainties.  This  first  effort  was  so  well  received  by  the 
AAH  PM  that  he  is  now  asking  for  continuous  tracking  of  his 
program  by  VERT  simulation. 

In  support  of  PEWS,  the  Army  Electronics  Command  conducted 
an  analysis  of  several  configuration  options.  The  objective  of 
the  effort  was  to  determine  viable  alternatives  to  the  proposed 
configuration  and  to  quantify  their  coat  and  schedule  impacts. 

The  one  analysis  which  considered  technical  performance  was 
performed  by  the  Navy.  Their  effort  was  concerned  with  whether  to 
test  a certain  radar  in  a laboratory  environment  or  if  the  risks  were 
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great  enough  to  require  actual  flight  tests  aboard  an  aircraft. 
The  risks  were  related  to  the  amount  of  now  technology  Involved. 


This  particular  analysis  was  conducted  "after  the  fact."  The 
decision  had  already  been  made  and  the  analysis  was  conducted 
as  their  first  attempt  to  use  VERT.  When  provided  the  results 
the  PM  believed  the  analysis  would  have  provided  him  valuable 
information  in  preparing  to  make  the  decision. 

In  two  other  applications  encountered  VERT  was  used  to 
evaluate  optional  approaches  to  satisfy  a requirement.  The 
option  of  an  automatic,  general  purpose  test  equipment  was 
compared  to  special  purpose  test  equipment  in  one  case.  In 
the  other  case  the  options  of  operating  a depot  activity  with 
contractor  personnel  was  compared  to  the  use  of  military  personnel. 

Although  other  applications  of  VERT  exist,  the  above  examples 
reflect  the  broad  range  of  applications  possible  with  this  j 

technique.  | 

Problems 

Several  comments  were  obtained  from  the  interviewees,  which 
Indicated  some  minor  problems  or  needs  for  improvjment  had  been 
encountered  in  their  particular  environments.  None  of  these 
problems  are  considered  major  impediments  to  the  acceptance  or 
effective  utilization  of  VERT.  In  fact  many  of  those  comments 
have  been  provided  Mr.  Moeller,  the  author  of  VERT,  who  has 
already  incorporated  many  changes  in  his  program.  In  other  cases, 
the  users  have  made  local  modifications  to  solve  their  particular 
problem.  One  agency  has  modified  the  program  to  provide  graphic 
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displays  of  tha  networks  and  plotter  outputs  (12).  Another 
agency  has  modified  their  input  format  to  make  VERT  more 
accessible  to  personnel  without  programming  experience  (ll). 

The  moat  frequently  addressed  problem  was  not  related  to  the 
VERT  program  Itself  but  was  concerned  with  the  collection  of 
data  needed  to  describe  the  probablistlc  behavior  of  the  variables 
(tine,  coat  and  technical  performance).  Although  VERT  allows 
the  use  of  many  different  distributions,  in  moat  previous  cases 
the  best  distributions  used  were  triangular.  This  procedure 
doesn't  really  use  the  capability  of  the  model  and  reduces  the 
accuracy  of  the  simulation  results. 

Another  problem  related  to  the  data  is  not  obtaining  ac- 
curate estimates  of  the  time  and  cost  from  the  ''experts."  Data 
frequently  had  to  be  rechecked  several  times  to  insure  its 
validity.  Data  sources  (experts)  have  consistently  given  overly 
optimistic  estimates.  In  some  cases  this  has  persisted  until 
the  data  sources  have  understood  the  data  was  to  be  used  In  a 
VERT  simulation  only  and  would  not  appear  in  other  documents 
which  could  cause  them  embarrassment. 

Although  not  a VERT  problem,  the  fact  that  VERT  is  not 
frequently  used  could  imply  problems  exist.  However,  there  are 
several  plausible  reasons  for  this  lack  of  use  other  than  a 
weakness  in  VERT. 

1.  The  VERT  model  has  only  been  available  for  about  I4.  years. 
Experiences  with  and  documentation  of  its  use  has  been  limited. 
Consequently,  program  management  office  personnel  are  unaware 
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of  its  existence  and  potential  applications  in  the  program 
development  environment. 

2.  The  time  a PM  allows  a systems  analysis  office  to 
produce  a risk  analysis  is  often  too  short  to  allow  the  set 

up  of  the  program  and  the  development  of  probability  estimates. 

The  actual  amount  of  time  required  is  related  to  the  complexity 
of  the  situation  being  modeled  and  the  availability  of  related 
historical  data. 

3.  Many  Army  organizations  who  provide  VERT  analysis  sup- 
port have  been  reorganizing  the  past  few  years.  As  a result,  , 

they  have  had  little  time  to  try  new  techniqviea  or  to  educate 

I 

program  office  personnel  about  the  capabilities  of  this  now 
technique, 

I4..  VERT  has  not  been  taught  or  publicized  to  any  significant 
degree  within  the  System  Acquisition  Community.  However, 
recently  the  Army  Logistics  Management  Center  (Pt,  Lee,  Va. ) has 
incorporated  some  discussion  of  VERT  in  their  advance  risk 
analysis  course  (l^)* 

Results 

VERT  simulation  results  provide  the  user  a considerable 
amount  of  information  describing  the  possible  real  world  impacts 
resulting  from  risk  situations. 

The  results  of  the  Cannon-Launched  Guided  Projectile  pro- 
grams' VERT  simulation  (mentioned  earlier)  provided  some  insight 
to  potential  future  actions.  The  simulation  indicated  that 
there  was  a 955^  chance  of  at  least  one  LRIP  manufacturer  qualifying 
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for  full  production.  It  also  indicated  the  total  cost  of  the 
program  would  bo  about  $9  million  over  baseline  cost  with  a 
9 month  (905^  confidence  level  was  * 10  months)  stretched  out  in 
the  schedule  (l:l4.)* 

In  the  self-propelled  howitzer  simulation  the  results 
indicated  IOC  would  be  achieved  13i^  months  (90^  confidence 
level  was  i 1 month)  after  initiation  of  the  effort.  This 
indicated  about  a 1 month  delay  in  the  planned  IOC.  The  major 
risk  determined  from  analysis  of  the  results  was  the  timely 
commitment  of  funds  (9:11). 

A final  example  of  VERT  simulation  results  is  provided  by 
the  configuration  management  analysis  conducted  in  support  of 
PEWS.  Several  of  the  alternatives  to  the  PEWS  configuration  of 
sensors  were  considered  feasible.  One  system  component  however 
was  considered  a high  risk  item  due  to  a design  improvement 
which  was  required.  This  improvement  required  a high  coat 
investment  and  a l6-month  delay  in  IOC  for  anticipated  marginal 
performance  improvements.  The  analysis  indicated  the  item 
could  be  replaced  by  another  item  with  minimum  impact  on  PEWS 
effectiveness.  One  additional  point  highlighted  by  the  analysis 
was  the  savings  of  $1.5  million  by  selection  of  one  of  the 
feasible  alternatives. 

Every  interviewee  stated  the  PM  they  supported  was  satisfied 
with  the  analysis  provided  by  the  VERT  simulation. 

By  his  actions,  one  PM  has  indicated  a hig^  degree  of  con- 
fidence in  the  use  of  VERT,  He  is  now  having  the  systems  anal- 


change  the  risk  situations,  he  will  be  provided  new  assessments 
of  risk  and  probable  impacts  of  any  decisions  ho  makes.  Be- 
cause of  this  PM's  demonstrated  interest,  this  systems  analysis 
office  plans  to  expand  their  VERT  efforts  to  other  programs. 
They  are  also  developing  an  education  program  to  be  used  to 
train  program  office  personnel  to  set  up  VERT  networks  and  to 
analyze  VERT  simulation  results.  Finally,  this  office  is 
preparing  a "paper"  to  present  at  the  Army  Operations  Research 
Systems  Analysis  Symposium  this  fall.  The  "paper"  will  address 
their  use  of  VERT  to  track  a development  program  (L3). 

The  discussion  in  this  chapter  shows  the  diverse  types 
of  risk  situations  which  can  be  analyzed  using  the  VERT  model. 
It  seems  apparent  from  the  interviews  conducted  that  the  VERT 
model  can  be  a very  beneficial  tool  for  program  management  once 

personnel  are  aware  of  its  availability  and  capability, 

t 
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CHAPTER  IV 


CONCLUSIONS  AND  RECOMMENDATIONS 


Conclu3ion3 

Although  only  a limited  survey  was  conducted , it  included 
four  different  Army  Commands  which  provide  functional  support 
to  many  of  the  Army’s  PM's.  Baaed  upon  these  informal  comments 
and  some  documented  VERT  applications  a few  general  comments  can 
be  made. 

1.  The  systems  analysts  have  found  VERT  to  be  a flexible 
and  potentially  powerful  tool  for  the  conduct  of  risk  analyses. 

All  the  personnel  interviewed  felt  VERT  was  the  best  tool 
available  to  solve  complex  risk  problems.  They  have  found  VERT 
programs  easy  to  set  up,  run  and  analyze.  Even  though  some 
users  have  made  modifications  to  the  program,  in  general  they 
feel  the  program  has  sufficient  programming  capabilities. 

2.  Program  Managers  seem  to  be  satisfied  with  the  results 
of  rioks  analyzed  using  VERT. 

3.  VERT  appears  to  be  well  suited  to  analyzing  risks  involv- 
ing time  and  schedule.  It  also  appears  to  have  great  potential 

to  solve  technical  risk  problems,  but  experience  in  this  area 
has  been  so  limited  that  no  conclusion  can  be  made. 

4.  Present  users  of  VERT  are  not  obtaining  maximum  benefit 
from  the  model  because  they  normally  use  triangular  (Beat,  Worst 
and  moat  likely  estimates)  distributions  to  describe  variable 
probabilities.  Apparently  this  is  caused  partly  by  the  lack  of 
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time  available  to  develop  more  accurate  depictions  of  the 
variables  and  partly  by  the  lack  of  experience  in  new  areas 
of  technology,  V/hen  working  in  these  new  areas  where  his- 
torical data  is  not  available  the  triangular  and  uniform 
(equally  probable)  distributions  may  be  the  best  estimates 
possible. 

VERT  can  be  used  to  analyze  a wide  range  of  risk 
situations.  It  has  been  used  to  make  decisions  involving 
hardware  testing,  low  rate  initial  production  and  configuration 
management.  VERT  analyses  have  been  conducted  on  both  simple 
and  complex  networks.  However,  when  the  networks  become 
extremely  complex,  it  is  usually  appropriate  to  develop  sub- 
networks, solve  these  independently  and  then  use  these  outputs 
as  inputs  to  the  final  general  network. 

Recommendati ons 

In  the  hope  of  improving  the  understanding  of  VERT  within 
the  Acquisition  community,  the  following  recommendations  are 
made. 

1.  An  information  paper  should  be  developed  and  disaeml- 
nated  to  all  Acquisition  Agencies  explaining  the  benefits  of 
VERT  and  identifying  activities  providing  VERT  risk  analysis 
service. 

2.  As  the  focal  point  of  Systems  Acquisitions  knowledge 
for  the  services,  DSMC  should  incorporate  into  its  curriculum. 
Instruction  concerning  VERT  as  a new  decision  analysis  tool  for 
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PM's,  Providing  this  instruction  should  serve  as  a strong 
stimulus  to  increasing  and  improving  the  use  of  VERT.  To  keep 
this  instruction  meaningful  the  college  should  develop  and 
maintain  a data  base  on  the  latest  applications  of  VERT  within 
the  acquisition  community. 

3.  A technical  reference  library  (file)  of  risk  analysis 
models  used  by  PM's  and  of  documents  discussing  their  applica- 
tion should  be  established  and  maintained  at  DSMC. 

4.  A profile  of  the  personal  reactions  of  PM's  to  VERT 
should  be  developed  (by  interviews)  and  analyzed  to  estimate 
the  actual  benefit  being  achieved  by  using  VERT.  This  investi- 
gation should  include  determining  if  the  PM’s  made  the  decision 
indicated  by  their  VERT  analysis.  It  should  also  be  ascertained 
to  what  degree  subsequent  occurrences  substantiated  the  model's 
predictions . 

5.  A representative  from  DSMC  should  attend  the  next 
OR/SA  Society  convention  (Pall  1977)  to  obtain  information  from 
a paper  being  presented  on  the  use  of  VERT  as  a program  tracking 
tool.  This  recommendation  is  supportive  of  recommendation 
#2. 

6.  To  improve  the  efficacy  of  the  VERT  model  in  support 
of  Acquisition  decisions,  system  analysts  should  be  encouraged 
to  develop  more  representative  probability  distributions.  At 
a minimum,  parametric  estimates  based  on  historical  data  can 
be  developed  at  an  acceptable  cost.  Implementation  of  recom- 
mendation #2  will  be  helpful  in  this  effort  by  impressing  upon 
PM  personnel  the  importance  of  using  the  best  estimates  that 
can  be  reasonably  developed. 

23 

[/ 

L 


r 


BIBLIOGRAPHY 


1.  Besson,  James  B.,  Risk  Analysis  of  the  Cannon 

Launched  Guided  Projectile,  Rock  Island,  Illinois; 

U,S.  Army  Armament  Command,  1976. 

2.  Blankenship,  Edward  DNC , Weapon  Systems  Analysis  Office, 

Naval  Weapons  Systems  Support  Activity,  Washington,  D.C. 
Telephone  interview  conducted  (Local  433-3621)  on 
k April  1977. 

3.  Bradley,  Paul,  DAC , Systems  Analysis  Office,  U.S.  Army  Tank 

and  Automotive  Command,  Warren,  Mi.  Telephone  intervie’f 
conducted  (Av  273-2379)  in  March  1977.  Mr.  Bradley  has 
used  VERT  in  support  of  the  XM-1  Tank  Program. 

I4..  Bevelhymer,  Herbert  L.,  A Proposed  Methodology  for  Weapons 

System  Development  Risk  Analyses,  Wright  Patterson  Air  Force 
Base,  Ohio:  Armed  Forces  Institute,  1973. 

5.  Piscilla,  Russell  R. , DAC,  Benet  Weapons  Laboratory,  U.S. 

Army  Armament  Command,  Watervilet,  N.Y.  Telephone  interview 
(Av  97(4.-5813)  conducted  30  Mar  1977.  Mr.  Piscilla  used 
VERT  in  the  development  of  a new  muzzle  break  for  Army 
artillery. 

6.  Knight,  James  R. , Comparison  of  PERT  and  PERT/Cost  with 

RISCA  and  RISNET,  U.S.  Army  Logistics  Management  Center. 


7.  Lenox,  Hamilton  T. , Risk  Assessment,  Wright  Patterson  Air 

Porce  Base,  Ohio:  Armed  Forces  Institute,  1973. 

8.  Moeller,  Gerald,  VERT  Documentation,  Rock  Island,  Illinois: 

U.S.  Army  Armament  Command,  l576. 

9.  Paarman,  Arthur  W.,  Risk  Analysis  of  the  MllOEl  Self- 

Propelled  Howitzer,  Rock  Island,  Illinois:  iJ.^.  Army 
Armament  Command,  1976. 

10.  Pearcy,  Stephen  B.,  DETRAK;  A Computer  Program  for  Evaluat 

ing  Decision  Trees,  Picatinny  Arsenal,  Dover.  K.J.. 


r 


11.  Pearcy,  Stephen  B.,  DAC , System  Evaluation  Office,  U.S. 

Army  Armament  Command,  Dover,  N.J.  Telephone  interview 
(Av  880-5765)  conducted  in  March  1977.  Mr.  Pearcy  has 
used  VERT  in  support  of  the  Cannon-Launched  Guided  Pro- 
jectile Program. 

12.  Reed,  Susan,  DAC,  Systems  Analysis  Office,  U.S,  Army 

Electronics  Command,  Pt,  Monmouth,  N.J.  Telephone 
interview  (Av  992-14,325)  conducted  in  March  1977.  Ms. 
Reed's  office  has  used  VERT  on  several  programs  to 
include  the  Platoon  Early  Warning  System, 

13.  Schapiro,  Paul  B.,  DAC,  Systems  Analysis  Office,  U.S,  Army 

Aviation  Systems  Command,  St.  Louis,  Mo.  Telephone 
interview  (Av  698-691I)  conducted  on  20  April  1977. 

Mr.  Schapiro  is  using  VERT  in  support  of  the  Advanced 
Attack  Helicopter. 

II4..  Toliver,  Donald,  Cpt. , USA,  U.S,  Army  Logistics  Management 
Center,  Pt.  Lee,  Va,  Telephone  interview  conducted 
(Av  687-l4.5'72)  in  Pebruary  1977.  Cpt.  Toliver  is  an 
Instructor  in  the  Risk  Analysis. 

15.  ■ . ■ * Evaluation  Review  Technique 

(y^RT)  Analysis  of~^e  Platoon  Early  Warning  System 
TPfiWS),  unpublished  report,  Pt.  Monmouth,  New  Jersey; 

U.3,  Army  Electronics  Command,  no  date. 


25 


r 

1 


/ 


VERT  PROGRAM  DESCRIPTORS 


1.  Some  distributions  available  to  describe  program  activities 
(processes ) 

a.  Histogram 

b.  Uniform 

c.  Triangular 

d.  Normal 

e . Gamma 

f.  Log  Normal 

g.  Gamma 

h.  Erlang 


2.  Outpt\t  options 

a.  Relative  Frequency  Distributions 

b.  Cumulative  Frequency  Distributions 

c.  Mean 

d.  Standard  Error 

e.  Coefficient  of  Variation 

f.  Mode 

g.  Measure  of  Kurtosis 

h.  Measure  of  Skewness 


i.  Exponential 
J,  Chi  Square 

k.  Beta 

l.  Poisson 

m.  Pascal 

n.  Geometric 

o.  Binomial 

p.  Hypergeometric 
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